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1. INTRODUCTION 

Indonesia's location in equator gives advantages for developing the agriculture sector in which 
unfortunately still relies heavily on conventional agriculture. The agriculture sector suffers the issue of old 
farming. This term refers to a lack of youth motivation to work in agriculture. The youth in the village would 
instead pursue their fortune in the city rather than continue their parents' footsteps to become farmers. 
This issue is not happening in Indonesia only. The lack of workers in the agriculture industry can be 
overcome by applying automation or digital farming. Robot technology is implemented to support farmers 
from seeding, farm maintenance, harvesting, and packing the agriculture product [1-7]. Automation or digital 
farming promises better and more efficient handling and treatment of the product [8]. 

The current application of robots for agriculture products is during harvesting [1-7]. This robotics 
application can be extended to sorting the agriculture product during packaging [8], where the robot is 
designed to grade the fruits and sort them based on color and size. This system can be realized by employing 
a camera functioning as the eye for the robot. The current camera comes in small size with enough pixels 
and quality to be the input for the robot system [1-7]. The small size camera can be attached to the robot as 
an eye in hand robot and attached somewhere else within the system [9-11] or as an eye to hand robot. 

The current application of the automatic sorting system is using an automatic belt conveyor or 
detection system by employing image processing for object detection; the detection can be a simulation [9] or 
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experimental prototype [10]. The objective of image processing is to separate the object from its 
background [11, 12]. Image processing is a powerful technique for detection in terms of grading and sorting 
agriculture production [9, 13-18]. Grading and sorting are similar in term deciding which one belongs to 
which [9, 13-18]. The common image processing techniques used are HIS [18] and HSV [19-21], L*a*b* 
histogram [22, 23], Otsu thresholding [20-22], edge detection [22, 24], image segmentation [6, 25], Blob 
analysis [9, 26, 27], random forest [28, 29], contour [29, 30], and convex surfaces [31]. The image processing 
is used to determine size [9, 10, 13, 18, 19], maturity [22, 29, 30, 32-34], bruising [35], color [9, 30] 
and pattern of the fruit [9, 17, 26, 30, 36]. The sorting process requires an additional sensor, such as a weight 
sensor to classify products based on the weight [10]. The researchers even include artificial intelligence to 
improve the effectiveness of the detection system [19, 32]. 

This paper presents the fruit sorting robot based on size and color applied for an agricultural product 
packaging system. The objective of this study is to design a sorting robot as the improvement of the current 
application of a belt conveyor or manual system for sorting the agricultural product. The main contribution 
and novelty of this project are creating a robot imitating the human’s arm for the sorting system. 
The application of a robot is expected to create a more efficient working environment. The camera applied is 
an IP camera to detect the fruit, and the image processing is kept simple to facilitate the limited source 
of microcontroller available in the market. The fruits to be sorted in this study are red and green tomatoes 
and purple/reddish and green grapes. The fruits are classified into small green and red, and big green and red. 
Experiments are conducted to show the effectiveness of the proposed method. 


2. RESEARCH METHOD 

This paper discusses the application of an arm robot manipulator as a fruit sorting robot based on 
color and size for an agricultural product packaging system. The object to be sorted considered in this study 
are red and green tomatoes and red and green grapes. The fruits are chosen since those fruits are a standard 
agricultural product. The proposed sorting robot system is shown in Figure 1. 
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Figure 1. Diagram block of the proposed method 


2.1. Mechanical and Electrical Design 

The robot applied in this study is a 4DOF arm robot manipulator where each joint is moved by 
a servo motor. The electrical and mechanical design is shown in Figure 2. This articulated robot is considered 
an ideal type to work as the sorting robot in which workspace is a crescent-shaped cross. 
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Figure 2. Electrical and mechanical design of the proposed arm robot, 
(a) Electrical design, (b) Mechanical design 
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The sorting robot system is equipped with a camera attached near to the initial fruit position and 
a proximity sensor attached to the end-effector to sense the distance between the end-effector and the fruit. 
The end-effector designed is a gripper to pick and place the fruit according to its size and color. The sensors 
applied are proximity and vision sensors. The proximity sensor attached to the gripper is the low-cost 
ultrasonic sensor (effectively from 2-400 cm). The vision sensor is an IP camera functioning as an eye to 
hand, which advantage is not suffering any occlusion while completing its task. The main controller is 
an ATMega2560 responsible for moving the servo motors, and the processor for image processing is 
Raspberry Pi. 


2.2. Fruit sorting algorithm design 

This study discusses the application of robotics to sort fruits based on size and colors. This 
application relies heavily on image processing to decide which fruit belongs to which container. The image 
processing is conducted in two steps, deciding the colors and recognizing the size. The color recognition is 
using HSV analysis, and the size recognition is conducted by calculating the diameter of the object/fruit in 
the grayscale image and setting the thresholding. 

The fruit color and size is conducted by converting the raw image captured by IP camera into 
a grayscale image, and isolate the object from the background by converting to a binary image, HSV 
analysis, setting the thresholding, and finally detecting edge to find the diameter of fruit by counting from 
the center to the edge. The first step of image processing is detecting the color of the fruit. The color analysis 
is by converting the RGB image to HSV. The process is started by retrieving the red (R), green (G), and blue 
(B) on a scale of 0-1. To achieve scaling 0-1 in the binary image, the scale of each component is decided by 
taking 255; therefore, each for R, for G, and for B is 255. The new value of R, G, and B are given by: 


R=——,G=—~, and B = — (1) 


scale,’ scaleg’ scalep, 








The first step to decide HSV analysis is to get H (Hue) value, which represents the color type. The H is taken 
by taking the largest (Myq,) and the smallest (m,,;,) of R, G, and B values. The difference between 
the largest and the smallest value is called A. The H is normalized by adding either 0, 2, or 4 to get any real 
number. Any arbitrary number below O and above 6 is considered redundant. The H value is decided by: 


undefined, if A= 0 
-B 


SE if Mmax = R 
H= BOR eit , and H’ =H x scaleņ, (2) 
<— if Max = B 


where Mmax = Mmax(R, G, B), Mmin = min(R, G, B), and A= Mmax — Mmin. 

The brightness of the assigned object is defined by the V (Value) value, which ranges from 0-255, 
where O is the darkest, and 255 is the brightest. S (saturation) gives the vibrancy to the object color. 
The value range from 0 to 255. The V value is given by: 


V = Mmax, and V’ = V x scale, (3) 


where scale, is 255. 
The lower the saturation value, the more fading the image is. S is also defined as the difference 
between the most significant and smallest channel values and divided by V, as shown below: 


0, ifV=0 
S = la, A E and S = S x scale.g, (4) 


where scale, is 255, and if V is 0, then S is 0. 

Therefore, as the IP camera in the packaging system captures the RGB images from the online 
video, the Raspberry PI converted those captured images into HSV value, as calculated above. The HSV 
values are used to determine the x-y coordinates position of the object and decide the color of the object 
between red and green. The red detection is for reddish color, starting from orange to very dark red. 

The next step of fruit detection is the determine the size of the fruit, to differentiate between red 
tomato and red grape, and green tomato and green grape. The size is decided by using ellipsoidal shape 
achieved from the grayscale image where the 4 ROIs (region of interest) of the detected fruit are identified. 
The thresholding value is set, and the edge of the object is detected. This process is as shown in Figure 3. 
The center of the fruit is determined by: 
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m m 
Cy =—, and cy=—, (5) 
Moo Moo 


where c, and cy is the center of detected fruit, and m,, is the geometric moment (p,q = 0,1,2,...). 
The geometric moments are calculated by the summation extends over all the elements in the detected fruit 
(£), and given by: 


eg evs, (6) 
while the diameter of the fruit is: 


earners: D 


Moo 


D = 


The diameter of the fruit gives the size of the fruit. As the image processing process is complete, 
the color and size of the fruit are known. The data is inputted to the microcontroller to sort the fruit by 
moving it to the box of red/green tomato or red/green grape. The result of image processing is shown in 
Figure 3, where the detected fruit is classified as a red tomato. 
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Figure 3. Red tomato detection using the proposed image processing method, 
(a) The detected red tomato, (b) The processed image 


3. RESULTS AND DISCUSSION 

This paper studies the image processing for fruit sorting robot based on the color and size. 
The applied robot is a 4DOF articulated robot with gripper as the end-effector. The HSV analysis decides 
the color, and thresholding and edge detection give the size of the fruit. The considered fruits are red 
and green tomatoes and red and green grapes. An experimental testbed is set, as shown in Figure 4. 
The experiments are conducted to show the effectiveness of the proposed method. The arm robot sorts 
and places the fruit in box | to box 4 according to the color and size. The first box is for red tomatoes, the 
second box is for green tomatoes, the third box is for red grapes, and the fourth box is for green grapes. 
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Figure 4. Experimental setup 
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3.1. Fruit detection 
The first experiment is setting the fruit in x and y coordinates, and the processed image is shown in 


Tables 1 and 2, where Table 1 is the data for tomatoes detection, and Table 2 is for grapes detection. The detection 
shown in Table 1 is only for red tomato, and Table 2 is green grape. The image plane considered in this study 
is 640x480 pixels. The fruit position is divided into the middle, right, and left area of the image plane, 
as shown in Tables | and 2. These x-y coordinates are necessary to tell the robot where to pick the fruit, 
and the size is necessary for the robot to place the fruit in the right assigned box, as shown in Figure 4. 
The circle is to show the viewer of the detected fruit, and it is achieved from the edge detection method. 


Table 1. Red tomatoes detection 











No Detected tomato Processed image X and Y-coordinates Fruit position 
1 a "m X= 323 In the middle of 
Y= 248 the image plane 
2 X= 84 In the right area 
Y= 228 of the image 
plane 
3 X= 508 In the left area 
Y= 244 of the image 
plane 
Table 2. Green grapes detection 
No Detected grape Processed image X and Y-coordinates Fruit position 
sii = z 
1 X= 361 In the middle of 
Y=312 the image plane 
In the right area 
7 X= 70 of the image 
Y= 301 plane 
zess he 
Y= 318 7 


plane 








3.2. Arm robot motion in sorting the fruit 

The arm robot moves to pick and place the fruit in its assigned box according to the location 
information shown in 3.1. The process of picking and placing is conducted by controlling the angle 
of the servo motors’ motion. Table 3 shows the angle of each servo motors during a particular motion. 
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Table 3. Servo motors angle according to robot motion 


Robot motion Angle (°) 

Servo motor 1 Servomotor2  Servomotor3 Servo motor 4 

(base) (link 1) (link 2) (gripper) 
Standby in the idle position 150 90 90 70 
Standby to pick the fruit 110 90 90 70 
Going down to pick the fruit 110 70 130 0 
Going up taking the fruit 150 60 130 0 
Moving the fruit to box 1 110 90 90 70 
Moving the fruit to box 2 75 90 90 70 
Moving the fruit to box 3 40 90 90 70 
Moving the fruit to box 4 20 90 90 70 
Placing the fruit to box 1 110 70 130 70 
Placing the fruit to box 2 75 70 130 70 
Placing the fruit to box 3 40 90 90 70 
Placing the fruit to box 4 20 70 130 0 


The screenshots of robot motion while sorting the tomatoes and grapes are shown in Figure 5, 
according to the experimental setup in Figure 4. The fruits are placed accordingly to the assigned boxes. 
The experiment in Figure 5 is conducted ten times for each fruit, as shown in Table 4, and Table 5 shows 
the time requires to pick, place the fruit, and the arm robot returns to stand by position in three experiments 
for each fruit. The robot is powered by a 2200 mAh battery to enable the robot working for 2 hours. 
The picking and placing fruit process is prone to illumination that can affect the performance of the robot. 
The “Vv” in Table 4 is when the robot was able to sort the fruit, and “X” is when the robot failed. 
The effectiveness of the proposed method in sorting, picking and placing the fruit 1s 80% for red tomatoes, 
90% for green tomatoes, 70% for red grape, and 60% for green grapes. 





Figure 5. Screenshot of robot sorting fruit based on size and color 


The failures are caused by the size of the fruit and illumination during the experiment. In experiment 
4 and 6 of red tomatoes, the tomatoes are too big for the gripper, and experiment 1, 4, 7, and 8, the grapes are 
too small for the gripper. Experiment 2 for green tomato is due to its changing color from green to red; 
therefore, the robot cannot decide whether the fruit is green or red. Experiment 4, 6, and 8 for red grapes, 
the color of grapes is too dark. The average time for the robot to accomplish sorting red tomatoes is 11.91 s, 
green tomatoes is 11.76s, red grapes is 12.56 s, and green grapes is 12.92 s. This is the time requires for 
the robot from detection in standby position (initial position), picking up the fruit, and placing it to 
the assigned boxes. The time requires is not much different in real-time. 

Table 4. Hit rate of fruit sorting, picking, and placing experiment 
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experiment Red tomato Green tomato Red grape Green grape 


n” 

1 X 
2 y X y y 
3 y y y y 
4 X y X X 
5 y y y V 
6 X y X y 
7 V y y x 
8 y y X X 
9 V y V y 
10 y y V y 
Total hit rate 8 9 7 6 
Total 30 of 40 (75%) 


Table 5. Time requires to picks, places the fruit, and returns to stand by position 


Fruit Exp-to Time (s) 
From stand by to From standby to placing From standby to placing the fruit 
picking the fruit the fruit to assigned box and return to stand by position 
Red tomato 1 3.67 9.80 12.46 
2 S12 9.82 11.54 
3 4.12 9.35 11.72 
Green tomato 1 3.94 10.02 11.45 
2 4.21 9.67 12.02 
3 3.52 11.15 11.82 
Red grape 1 3.97 10.98 12.10 
2 4.32 11.76 12.56 
3 4.67 11.82 13.01 
Green grape 1 4.12 12.01 12.99 
2 3.98 11.78 13.22 
3 4.23 12.13 12.54 


4. CONCLUSION 

This paper discusses the application of an arm robot as a sorting robot based on color and size. 
The image processing conducted by setting HSV value analysis to recognize the color of the fruit and the size 
is achieved by calculating the diameter of the fruit in the grayscale image and setting the thresholding. 
The fruits to be sorted are red and green tomatoes and red and green grapes. The experiment was conducted 
ten times for each fruit. The effectiveness of sorting, picking, and placing the fruit is 80% for red and 90% 
for green tomatoes, 70% for red, and 60% for green grapes. The failures are due to the size of fruits 
and illumination during the experiment. This drawback can be overcome by applying flexible gripper that can 
adjust to the size of the fruit, and improving the image processing algorithm to be less prone to illumination. 
The time requires to accomplish the fruit sorting is noted for three experiments for each fruit, and the average 
time for the robot to sort red tomatoes is 11.91 s, green tomatoes is 11.76 s, red grapes is 12.56 s, and green 
grapes is 12.92 s. The experimental result shows that the arm robot is applicable for a fruit sorting robot 
based on the proposed method in this study. 
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